A Parsing Method for Identifying Words in Mandarin Chinese Sentences

T h i s paper presents a pars ing m e t h o d for i den t i f y ing words in m a n d a r i n Chinese sentences. T h e iden t i f i cat ion system is composed of a T o m i t a ' s parser augmented w i t h tests o r i g ina l l y a pa r t o f the Engl ishChinese machine t rans la t i on system C C L E C M T together w i t h the associated augmented contextf ree g r a m m a r for w o r d compos i t i on . T h e s imple augmented g r a m m a r w i t h the score f unc t i on effectively captures the i n t u i t i v e idea of longest possible compos i t ion of Chinese words in sentences a n d , at the same t i m e , take i n to cons iderat ion the frequency counts o f words. T h e ident i f i ca t ion rate o f th is system for the corpora taken f r o m books and a newspaper is 99.6%. T h i s iden t i f i ca t ion system is s imple , bu t the iden t i f i ca t ion rate is re la t ive ly h i g h . T h e m i n i m u m element for word -compos i t i on pars ing is down to characters as opposed to sentence pars ing down to Chinese words. I t has the po ten t i a l o f incorpor a t i n g phrase s t ructures and semant ic checking in to the sys tem. In th is way, wo rd iden t i f i ca t ion , syntact ic and even semant ic analysis can be organized in to a single phase. T h e results of tes t ing the word ident i f i ca t i on on co rpora taken f r o m books and a Chinese newspaper are also presented. 1 In t roduc t ion